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PREFACE 



The Research on Evaluation Program is a Northwest Regional 
Educational Laboratory project of research, development, testing, 
and training designed to create new evaluation methodologies for 
use in education. This document is one of a series of papers and 
reports produced by program staff, visiting scholars, adjunct 
scholars, and project collaborator s—all members of a cooperative 
network of colleagues working on the development of new 
methodologies* 

What kinds of cost analysis studies are being done in educational 
evaluation? The educational literature shows very few 
applications of cost methods, especially the seemingly most 
appropriate method: cost effectiveness analysis. The health 
literature shows a greater use of cost methods, but for treatment 
or intervention research rather than for program management 
purposes as in education. An examination of a contracting firm's 
cost studies revealed the use of only the simplest cost methods 
and those for management purposes. A national study of SEA 
evaluation units showed an increasing mandate for the use of cost 
methods and a movement toward the use of somewhat more complicated 
procedures. An interpretation of these findings using knowledge 
transfer theory shows that practitioners have little problem with 
the credibility and relevance of cost methods, but major 
difficulties in understanding and implementing them. 



Nick L. Smith, Editor 
Paper and Report Series 
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COST ANALYSIS IN EDUCATIONAL EVALUATION 
Introduction 

Thare ace a number of arguments why evaluators should be 
using cost analysis methods in their work. Indeed, Wortman 
(1983) has gone so far as to state that "... in conclusion, 
knowledge and use of CEA/CBA [Cost Effectiveness Analysis/Cost 
Benefit Analysis] may be essential for the very survival of 
evaluation research" (p. 256). In spite of such urgent appeals, 
the actual use of cost analysis methods has apparently been slow 
to occur, especially in educational evaluation. Since we knew of 
no formal study of the practice of cost analysis in educational 
evaluation, we conducted a series of exploratory studies to 
identify the types of cost work being done and to study the 
impediments to the use of cost procedures in educational >, 
evaluation. 

The purpose of this article, then, is to provide evidence on 
the amount and types of cost analysis activities currently done 
in educational evaluation. We seek to provide a "state of the 
practice" report on the use of cost methods in conducting program 
and intervention evaluations. By way of introduction, we briefly 
review the arguments for using cost methods in educational 
evaluation, we then outline four basic types of cost methods, and 
then we summarize the evidence of their use as recorded in the 
educational literature. 

The main body of this article reports the results of three 
studies undertaken to document the actual use of ost methods. 
The first study consisted, of a review of cost studies done 



in health evaluation. Since cost methods have been used more 
extensively in health evaluation than educational evaluation, 
this review established a benchmark for comparison and helped us 
to identify what to look for in reviewing educational uses of 
cost methods. Next, in order to become familiar with the nature 
of cost studies as actually practiced, we examined all cost 
studies (15) conducted by an educational research and development 
firm over a five-year period (1977-1982). This was study two, 
which, along with the first study, prepared us to conduct the 
third investigation — a national survey of all evaluation units in 
state departments of education. This third study provided 
comprehensive information on current practice and future demands 
for cost work at the state level. The results of these three 
studies reveal that although some cost work is being dona, it is 
simpler and more descriptive than the sophisticated, comparative 
approaches advocated in the literature. 

In concluding t < .s report, we review the results of the three 
studies in terms of knowledge transfer theory to explain why some 
of the impediments to the use of cost procedures may be arising 
and to suggest ways in which these impediments may be overcome. 
Before proceeding to these three studies, however, we provide 
some introductory background. 

The Need for Cost Methods 

Although during the 19603 and early 1970s financial resources 
for education expanded rapidly, a number of factors have combined 
to change the outlook for the 1980s. Declining tax bases, tax 
expenditure limitations, school bond a-id levy failures, continued 
inflationary pressures, and reductions in the level of state and 
federal aid have combined to place tremendous stress on 
educational systems. Evidence indicates that this trend will 
continue. Resource projections (e.g., Kirst and Garms, 1980) 
sugoest that educational revenues will not keep pace with 
inflation in the 1980s. 
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Educational personnel struggle to cope with these budgetary 
problems. Budget reductions have to be made under a variety of 
constraints , including not violating employee contracts , not 
alienating political interest groups , and not lowering the 
quality of education provided. Further, restrictions on, or 
prohibitions against, deficit spending by school districts place 
additional pressure on school personnel to identify where budget 
reductions can be aide with minimum detrimental impact. Even 
when budget reductions are not an issue, the public has 
increasingly demanded analyses of educational costs because the 
tight economic times have heightened public concern for financial 
accou ntability. 

One possible way to respond to these budgetary problems is to 
provide information on the relationships of program costs to 
program outcomes. Historically, educational evaluators have 
studied program operations (process) or impact (outcomes) but 
have not related that information to program costs. They could 
say which program alternative was more effective, but did not 
know whether the increased effectiveness was related to increased 
cost. Budget managers historically have made recommendations 
based on comparative costs, but could not say whether cheaper 
alternatives were more or less effective. Only by incorporating 
both costs and outcomes within comparative studies of program 
alternatives can one reliably determine which alternative is most 
effective for a given cost, or how much it would cost to obtain a 
desired level of effect. 

Cost and effects data must, therefore, be combined in the 
same analysis in order to answer crucial administrative questions 
such as: What is the total cost for achieving a certain level of 
effectiveness? What is the average cost per unit of 
effectiveness? What is the marginal cost for additional units of 
effectiveness? (Levin, 1975), 

The general decline in educational resources is having a 
definite impact on the practice of educational evaluation, and 
appears to be increasing evaluators* concerns with cost analysis 
methods. A ree nt study (Gray, Caulley, and Smith, 1982) of the 



impact of the national Education Consolidation and Improvement 
Act of 1981 on evaluation practice at the state and local level 
found that declining budgets were in fact having a greater 
influence on evaluation activities than the new federal 
legislation. Reductions in Chapter 1 budgets have resulted in 
major reductions in instructional staff and in the numbers of 
students served. Under the new block grant legislation 
(Chapter 2), most large school districts report a reduction in 
funds and the need to identify methods for selecting among 
programs competing for increasingly scarce resources* Evaluators 
doing contract service work with local districts and state 
departments also report an increase in the number 6f cost 
analysis questions from their educational clients (Gray and 
Smith, 1983). 

The relationships between program costs and program effects 
has thus become increasingly important. Research on educational 
innovation has repeatedly shown that one prerequisite to 
successful change is the presence of locally felt need (Berman 
and McLaughlin, 1975; Cheever, Neill, and Quinn, -1976; Fullen 
and Pomfret, 1975). At present there appears to be some 
evidence of locally felt need for greater use of cost analysis 
techniques in educational evaluation. 

Types of Cost Methods 

A variety of methods could be used for dealing with cost 
problems in educational evaluation. Operations research 
techniques are available for minimizing costly program features 
such as student transportation and the inefficient use of school 
facilities, and to define program configurations that will 
minimize costs and maintain needed services (e.g. , Page, 1979; 
Wholeben, 1982). Management consultation procedures are 
available for considering changes in programmatic functions when 
resources dwindle (e.g., Stanfield, 1982). Applied economists 
have argued, however, that the methods of greatest analytic 
usefulness are likely to be cost-benefit, cost-effectiveness, and 
cost-utility analyses which enable one to combine outcomes with 
costs to determine the most effective and efficient program 
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options. Cost-feasibility analysis can also be a useful planning 
method, although it does not combine outcomes with costs. 

Levin (1975, 1981, 1983) provides an excellent introduction 
to the use of these four methods in evaluation. Cost-feasibility 
analysis involves estimating whether the cost of a program 
alternative qualifies it as a real possibility within existing 
financial resources. One simply determines the cost of various 
components of the alternative and then compares the total cost 
with the amount of resources available from existing sources. 
Cost-feasibility does not deal at all with program outcomes, ar>d 
therefore provides no way of comb* ling costs and outcomes in nn 
overall analysis. The other three techniques do allow one tc 
combine both costs and outcomes in a single analysis. 

Cost-utility analysis involves the subjective measurement of 
probable outcomes, such as when an administrator or school 
committee estimate likely program effects. One can integrate 
multiple outcome measures into a single value, but because the 
measures and analysis are highly subjective, the results are 
generally not replicable. This form of analysis is useful as a 
planning tool for administrators, or as a device to aid in group 
discussion of possible program effects, but provides a weak basis 
for making programmatic decisions. 

Coat-benefit analysis provides replicable results and enables 
one not only to compare alternatives for a given program, but to 
compare across programs which have different classes of 
outcomes. Because all outcomes are expressed in terms of doller 
benefits, one can compare reading programs with counseling 
programs with athletic programs. One major problem with using 
cost-benefit analysis in programmatic evaluations, however, is 
that it is frequently very difficult to assign. monetary values to 
program outcomes. Although cost benefit analysis has 
traditionally been advocated as the method of choice because 
it enables multiple comparisons to be made, problems in 
monetizing non-marketed costs have led some writers to argue that 
it should no longer be used in certain policy are -s (Kelman, 
1981). In education, since dollar values assigr to such 
outcomes as increased music appreciation, reading comprehension, 
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or self-confidence are highly questionable, cost-benefit studies 
frequently have little credibility with local administrators. 

Cost-effectiveness analysis consists of representing program 
outcomes not in terms of monetary units, but in terms of other 
effectiveness units such as reading scores, attitude scale 
scores, and behavioral rates. Effectiveness units commonly 
consist of the standard. outcome measures currently used in 
educational evaluation./ Because one does not convert all 
outcomes to the same unit (dollars) , one cannot use 
cost-effectiveness analysis to compare programs of different 
types (e.g., to compare reading program^ to athletic programs). 
Cost effectiveness analysis does enable comparisons of programs 
with similar outcomes, however, such as which of two 
instructional strategies most effectively improve reading 
scores. Of the available techniques, therefore, 
cost-effectiveness analysis would seem to be the best method to 
use in most cost evaluations of educational programs. 

If cost-effectiveness analysis is the most desirable method 
for conducting cost studies in educational evaluation, to what 
extent is it currently being used? We turned to the educational 
literature in an attempt to answer this question. 

The Educational Literature 

In order to identify current practices in the use of 
cost-effectiveness analysis in educational evaluation, we 
searched for published examples of studies employing it. 
Initially, a search of all ERIC entries from 1977 to 1983 was 
made, followed by a search of the references cited in the 
obtained articles. 

From this search we found several articles and books that 
have been written primarily to urge evaluators to use 
cost-effectiveness analysis; some of these references also 
contain reviews of previous uses of cost methods. 

In 1970, Alkin advocated the use of cost analysis in the 
evaluation of instructional programs. Levin (1975) has provided 
an introduction to cost analysis as well as a brief overview of 
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cost-effectiveness and cost-bene: i — ds in studies of such 
educational topics as manpowet ura,in-r^ ncngrams, teacher 
selection; computer assisted instruction, dropouts, preschool 
programs, and compensatory education. Elsewhere, Levin (1981) 
critically examined seven illustrative studies of the use of 
cost-benefit and cost-effectiveness analysis in education using 
examples from many of the same areas. Levin (1983) and Thompson 
(1980) have also written texts on the use of cost-effectiveness 
and cost-benefit analyses in evaluation. One of the most recent 
additions to this literature is a volume on doing 
cost-effectiveness studies of educational evaluations themselves 
(Alkin & Solomon, 1983). Other articles (e.g., Lorenzen and 
Braskamp, 1978; Carr et al., 1982; Klees and Wells, 1983) have 
dealt with the problems and benefits of using cost-effectiveness 
and cost-benefit analyses in evaluation. 

Unfortunately, we found few examples of cost-effectiveness 
analysis used in an evaluation to make comparative judgments of 
program or intervention alternatives. We Were not searching for 
uses of cost-effectiveness analysis in educational finance, or in 
studies of general questions of educational policy. For example, 
we were not looking for such studies as Carney 9 s (1975) 
investigation of the cost-effectiveness of educational television 
for improving education in developing nations, but we were 
looking for the use of cost procedures to make program decisions 
at the administrative or managerial level. Most studies we 
identified fell somewhere between national policy studies and 
program management applications, although usually closer to the 
former than the latter. Pot example, Levin's (1970) 
cost-effectivenesa study of teacher selection and the Levin and 
Woo (1981) study of the costs of computer assisted instruction in 
compensatory education, we classified as mure policy than 
programmatic in focus. 

A good example of the type of cost-effectiveness application 
we were oearchino for is the Quinn et al. (1984) study of the 
cost-effectiveness of two alternative math programs for teaching 
fifth grade math. Quinn et al. used identical costs and effects 
measures to analyze the effectiveness of two in-place math 
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programs. In addition , they used a measure of social economic 
status to moderate for student differences. Their study is an 
excellent example of a full cost-effectiveness approach at the 
program level. 

In general, however, we found very few such studies. Some 
articles were erroneously titled; they did not actually deal with 
cost-effectiveness analysis. Other articles only discussed 
cost-effectiveness issues and did not present results, or they 
presented results based only on subjective judgments without 
empirical evidence. Other reviewers of this literature have 
found similar results. In reviews of compensatory education 
evaluations, for example, Mullin and Summers (1983) found that 
only 8 of the 47 studies they examined had looked at the 
relationships between costs and program effects. Wargo et al. 
(1972) had previously found 38 studies of compensatory education 
that claimed to look at cost-effectiveness issues, but 32 had 
done so on a purely subjective basis. We found very few good 
examples of cost-effectiveness studies of program alternatives or 
interventions in the educational literature. 

Perhaps we are simply looking in the wrong places, i.e., the 
formal journal literature on educational evaluation. It is 
possible that evaluators and administrators do cost-effectiveness 
studies, but for inhouse use only, and do not generally 
disseminate or publish their work. Clearly, there are 
innumerable examples of unpublished evaluations which study 
effects but pay no attention to questions of cost. One might 
argue that this is to be expected, given that most educational 
evaluators are trained as field researchers with little or no 
exposure to cost analysis procedures. 

On the other hand, we found examples of published studies by 
evaluators which looked at costs only, with no attention paid to 
effects. For example, Smith and Hendrickson (1982) report an 
evaluation of the nature and costs of a district school health 
services program. This evaluation contained no direct measures 
of the effectiveness or quality of the health care provided. The 
authors ci:e this lack of effectiveness data as a major weakness 
in their study. Anderson (1982) describes a study of the cost of 
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special education in the New York City schools, again with no 
mention of effects data, although such data would seem essential 
in interpreting the cost data for policy decisions. Hartman 
(1981) even developed a complete model for estimating the costs 
of special education programs, again with no mention of effects 
data. 

Pogrow (1983) rioints out that, historically, studies of 
school finance have\ focused only on educational costs, while 
evaluation studies h^ve focused only on effectiveness 
indicators. At the policy level, economists have attempted to 
combine costs and effects to develop educational production 
functions. Pogrow argue* that none of this work has produced 
consistent gains in educational ef fectiveness f and therefore that 
costs and the improvement cf efficiency should be of prime 
concern. 

Specifically, cost-effectiveness analysis needs to 
begin to treat effectivenes s as essentially a 
constant , and examine the effects of substantially 
varying the cost structure of education. This 
suggested shift in analytical emphasis derives from 
the belief that improving thu effectiveness of 
schooling is not a realistic interventionist policy 
goal for the eighties, and that the goal of 
maintaining existing levels of effectiveness at 
reduced cost levels (i.e., improving efficiency) 
should be substituted. 

(Pogrow, 1983, p. 75) 

As we shall discuss in the next section of this article, this is, 
in fact, the position taken by many researchers in conducting 
cost evaluations of health care programs. 

Having discussed the need for cost analysis procedures in 
educational evaluation and having xdentified the seemingly most 
appropriate method (cost-effectiveness analysis), we then 
proceeded to describe how this method had been used in the 
practice of educational evaluation as reflected in the 
professional literature. Unfortunately, our search of the 
educational literature turned up very few actual applications of 
this method. We, therefore, next turned our attention to a 
formal study of the health evaluation literature since we had 
observed that cost-effectiveness analysis seemfl to have been used 



more \ idcly there than in education. This review of the health 
literature provided valuable insights into our subsequent case 
study of cost analysis practice by a research and development 
firm and our national survey of cost analysis studies by state 
departments of education. 



Studies o2 Practice 

< 

Study 1: Coat Analysis in Health Evaluation 

In order to identify the evaluation uses of cost analysis in 
health areas, we first conducted a MED-LINE search of all 
articles between 1977?1983 and then searched the reference lists 
of the articles identified from the MED-LINE search. (See Smith 
and Smith, 1983, for a bibliography of over 350 references to the 
use of cost analysis in program evaluation in education and 
health. ) 

Cost-effectiveness and cost-benefit analyses began to be 
applied in health care evaluations in the 1960s. Since that 
time, the number of applications has steadily increased. Warner 
and Hutton (1980) report that a search of the literature for 
1966-1978 identified more than 500 references on cost-benefit and 
cost-effectiveness analysis, growing from an average of 17 per 
year in the early years to an average of 73 per year in the 
• latter four years. They indicate that the literature was growing 
more rapidly in medical journals than it was in non-medical 
journals and that proportionally more cost-effectiveness studies 
were published in later years than were cost-benefit studies. A 
similar increase in the number of cost studies of health care 
programs was also reported by Thompson and Fortess (1980) who 
reviewed the published literature from 1967 to 1978. They 
reported an average of seven published articles a year from 1967 
to 1972, and an average of 33.5 published articles a year from 
1973 to 1978. Because jbt the large number of cost studies 
conducted in the heai^h care field, this review will focus on 
categories of studies and will depict trends in the us 3 of cost 
methods in health evaluations. 
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Cost -effectiveness analyses in health care programming have 
used diverse measures of effectiveness. For example, mortality 
(Klarman, Francis 6 Rosenthal, 1968), morbidity (Rogers, Eaton & 
Bruhn, 1981), ard days of survival (Skellie, Mobley 6 Coan, 1982) 
have been used as measures of program effectiveness. Other 
measures of effectiveness have focused on program success in 
optimizing health status. Here effectiveness has been indicated 
by ccores on an activities-of-daily-llving scale, tests of mental 
status (e.g., Doherty & Hicks, 1977; Berwick 6 Komaroff, 1982; 
Weissert, Wan, Livieratoa 6 Katz, 1980) or behavioral measures 
such as weight loss (Yates, 1978). 

We categorized the literature on the cost-effectiveness of 
health care programs into three logical groups. The first 
category contained articles designed to increase the 
understanding of cost-effectiveness methods as applied to 
particular health care program content areas. Second, a series 
of instructional articles or "how-to guides" foi the conduct of 
cost-effectiveness analyses were identified. Finally, we found 
articles that discussed more abstract methodological and 
theoretical issues. Each of these categories of articles will be 
discussed below. 

In terms of program content areas, much attention has been 
paid to the difficulty of determining the cost effectiveness of 
(1) health prevention, and (2) alternative home care treatment 
programs. Several reviews of articles describing the 
cost-effectiveness of these two program areas have documented 
both the number of analyses as well as the quality of the 
analyses. 

Unfortunately, while many cost-effectiveness studies of 
prevention programs have been conducted, reviewers have been 
discouraged about the quality of these studies (e.g., Scheffler & 
Paringer, 1980; Snopard & Thompson,: 1979). For example, in 
1981, Rogers, Eaton, and Bruhn reviewed the published cost 
analyses of prevent ion programs from 1969 to 1979. Even though 
they were supportive of the utility of these analyses for program 
planning, they frequently pointed to a number of methodological 
weaknesses. Rogers et al's criticisms included the use of self 
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rather than random selection of program participants, the failure 
to use a control or comparison group, and the use of criteria 
that were not clearly or precisely measured. 

In regard to the cost-effectiveness of alternative home care 
treatment programs, one major review by Hammond (1979) points to 
an important and frequently encountered methodological problem. 
Hammond noted that most articles he reviewed assumed equal 
effectiveness across programs, and, as a result, no effectiveness 
data were gathered. Rather, the cost per patient was assessed 
and compared across programs. Thus, one could argue that many of 
these so-called cost-effectiveness studies did not warrant that 
label, since they did not involve the study of the interactions 
between costs and effects. This failure to measure ef fectiveaess 
is also found in other studies of the "cost-effectiveness" of 
health care programs (e.g., Dawson, Cohrs, Eversole, Frankenburg 
& Roth, 1976; Dawson, Cohrs, Eversole, Frankenburg & Roth, 1979). 
An additional weakness of these cost-effectiveness analyses of 
home care programs is pointed to by LaVor and Callender's (1976) 
criticism that such studies focused primarily on short-term 
results and were inappropriately generalized across hospital 
settings. 

Criticisms of cost-effectiveness studies in ether hoalth care 
areas have been posed as well. Jacobs and Chovil (1983) reviewed 
cost-effectiveness analysis of corporate medical care programs. 
They reviewed 11 articles: 4 using cost-effectiveness methods, 
and 7 using cost-benefit methods. Their conclusions were 
discouraging. They critiqued most studies as being 
methodologically weak; some failed to specify alternative 
programs, and in many studies, costs and benefits were 
inappropriately valued. 

Finally, cost-effectiveness analyses have also been conducted 
in mental health settings (e.g., Newman, Burwell & Underhill, 
1978; Longabaugh, 1979; Luft & Fakhouri, 1979; Fishman, 1979; and 
Yates, 1979; Saxe, 1980). In recent years, the National 
Institute of Mental Health has supported the application of cost 
analysis in mental health (e.g., Newman, Barwel*, and Underhill, 
1978; Sorenson and Grove, 1978, and Roid, Brodsky, and Bigelow, 
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1980) , and there has continued to be calls for the increased use 
of cost methods in this area (e.g., Kiesler, 1980, 1982). Many 
of these discussions in mental health have pointed primarily to 
the promise of cost-effectiveness analysis and have focused less 
on methodological weaknesses. 

Not surprisingly, and perhaps because of the criticisms of 
cost-effectiveness studies of health care programs, the medical 
literature contains many "how-to" articles. These articles tend 
to be content-area specific, and usually include a review of 
relevant literature along with a step-by-step procedural guide 
for the conduct of cost-effectiveness analysis for that 
respective content area. Such instructional articles are found 
in the areas of health service delivery (Dunlap, 1975) , 
rehabilitation (Schalock & Harper, 1982), prevention (Rogers, 
Eaton « Bruhn, 1981; Scheffler & Paringer, 1980; Shepard 6 
Thompson, 1979) and general health care programs (Weinstein 6 
Stason, 1977). 

Finally, there is also a body of published health literature 
that focuses on methodological and theoretical issues. For 
example, Doherty and Crakes (1980) point to the importance of 
differentiating between the costs of developing a health care 
program versus maintaining a program. Failure to take this / 
distinction into account can result in an overestimation of j 
program costs. Stoddart (1982) encourages the education of 
clients on cost methods as a means of increasing utilization of 
results. He contends that educating clients will result in 
greater impact on the decision-making process. In contrast, 
there are researchers who argue that the results of cost studies 
should be only a part of the decision process, since the methods 
are difficult, unreliable (Lave & Lave, 1978) and easily biased 
(Joglekar, 1982). 

While health applications of cost-effectiveness analyses are 
on the increase, the literature suggests that the quality of 
these applications is questionable. Further, there is now 
increasing question about the role of cost analysis res Its in 
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the decision-making process. These writers do not suggest 
abandoning the methods, however, but rather indicate a necessary 



period of questioning in the development of the techniques; 
criticisms and questions about their application suggest a 
positive maturation in the use of cost analysis methods in health 
care evaluation. That is, the initial enthusiasm is being 
replaced by the critical evaluation needed for refinement of the 
techniques. 

In education, the work on educational production functions 
reflects an attempt to combine costs with effects at the policy 
level. There has been much less effort to combine the two at the 
managerial or administrative level, however. This has not been 
the case in health evaluation where a far greater number of 
cost-effectiveness studies of program or treatment alternatives 
has been performed. There have been a variety of methodological 
flaws in many of these studies, however, and many have not been 
complete cost-effectiveness studies. Some evaluators in the 
health profession have apparently anticipated Pog row's (1983) 
analysis, as previously mentioned. For example, in a review of 
14 health- evaluations citing cost-effectiveness techniques, 
Hammond (1979) found that most studies assumed that program or 
treatment alternatives were equally effective, and so studied 
only costs in order to increase efficiency. 

In summary, cost analysis, particularly cost-effectiveness 
analysis, is much more prevalent in health evaluation than in 
educational evaluation, especially at the program or intervention 
level. Although many of the health studies have methodological 
shortcomings and fall far short of following complete 
cost-effectiveness procedures, analysts continue to argue the 
need for them and researchers have managed to overcome many of 
the problems of implementation. 

Why is the situation in educational evaluation different? 
There appears to be a need for cost analysis, but perhaps that 
need is more recent than in health, or perhaps educational 
researchers have lacked the training and experience to implement 
such studies. Our experience suggests that there is some truth 
in each of these explanations, but that more important reasons 
why cost analysis is not widely prevalent in educational 
evaluation has to do with tb* purpose for which most educational 
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evaluations are conducted. Many of the cost studies done in 
health are conducted to evaluate treatment or intervention 
effectiveness, they evidence a strong "treatment research" 
orientation. Most program evaluations in education focus more on 
program management concerns and less on' treatment research. This 
difference in purpose suggests that different types of cost 
procedures may be needed. 

In our second study, then, we sought to identify more clearly 
the nature and purpose of the cost studies which were actually 
being done in education. We examined the cost-related studies 
conducted by a research and development firm as part of its 
educational field service contract work. The results of this 
study suggest that, in many cases, traditional cost analysis 
methods are inappropriate for managerial purposes in educational 
program evaluation settings. 

Study 2: Cost Analysis bv an 
Educational R&D Firm 

We identified and analyzed all cost studies (15) done over a 
five-year period, from 1977 through 1982, by a regional, private 
research and development agency as a part of its evaluation 
service work. These studies were conducted for school districts 
(local education agencies — LEAs) and state departments of 
education (state education agencies — SEAs) . (See Smith, 1983a, 
for a more complete description of this study.) 

The studies were identified by surveying agency staff 
concerning their own pas^ and current cost projects and by asking 
about other staff in the agency who might have been involved in 
such work. A copy of the published report of each cost study was 
abstracted according to a standard case study report form and an 
interview with each study's principal investigator was held to 
check the accuracy of the abstracted description and to collect 
information that could not be determined from the written 
report. The final list of abstracted 'cost studies was reviewed 
by agency administrators to insure that all major cost studies 
had been identified and properly described. 
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For each study , the following information was collected: 
report title, client, year of the study, budget allocated to the 
study, length of time taken to complete the study, and the length 
of the report. The studies ranged in cost from $1,666 to $75,000 
and took from 2 weeks to 36 months to complete. The reports ran 
from 5 to 350 pages. Four of the studies were done for LEA 
clients, 3 for SEA clients, and 2 studies were reported jointly 
to both LEA and SEA clients. The largest number of the studies, 
6, were conducted specifically for use by chief state school 
officers. 

Each cost analysis study was reviewed to identify the 
decision context, to describe the cost methods used, and to 
critique the application of cost procedures. The completed case 
studies were sorted into categories according to type of cost 
problem. Four categories were identified: (1) cost comparisons 
between two or more programs or entities, (2) cost description of 
a single program or entity, (3) budgets and planning, and (4) 
policy analyses. 

The first category consisted of cases which simply compared 
costs of alternative Programs or entities. In Case 1, the costs 
of three alternative pupil transportation systems were compared. 
Cases 2 and 3 compared costs of independent versus cooperative 
pupil transportation systems, although only costs for the 
cooperatives were delineated. Case 4 looked at the costs of 
compensatory education programs, while Case 5 looked at the cost 
of student activity programs. 

The second category cases contained descriptions of the costs 
of a given program or entity. Typically, studies of this sort 
were conducted to depict an existing program, and they included a 
description of the costs of running that program. Cases 6 and 7 
fell within this category. In thvise cases, the principal 
investigators had abstracted existing cost records and had 
tabulated the total cost of the program. Case 6 looked at the 
cost of educating special populations, and Case 7 looked at the 
cost of educating refugee students. Case 8 described a series of 
reports referred to as "adopters guides." These adopter's 
guides were provided to school personnel when a program was 
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proposed. Each guide described step-by-step procedures for 
setting up and running a program, and often included a 
rudimentary description of the expected costs the program would 
incur. Methods of cost analysis used in these reports range from 
a verbal discussion of costs to a detailed tabulation of program 
costs. 

The third category contained a variety of budget and Planning 
studies , each of which outlined a budget plan. Case 9 provided 
an expenditure analysis of state educational funds, which enabled 
chief state school officers to better explain school expenditures 
to taxpayers. A follow-up study analyzed state-by-state 
expenditures for 6 states. Case 10 described a four-step study 
that suggested changes in a state school finance formula. In 
Case 11, detailed budget formulas were developed for use in 
determining equitable distribution of funds for compensatory 
education programs. 

The fourth category consisted of policy analysis reports . 
The cases in this category provided, theoretical discussions of 
policy changes affecting school functioning. Case 12 described 
the effects of the Elementary and Secondary Education 
Consolidation Act of 1981 on SEA policy. The report provided 
chief state school officers with alternative methods for dealing 
with policy changes required by the act. Cases 13 and 14 looked 
at implications for school funding which resulted from the 
passage of Proposition 13 in California and the One Percent 
Initiative in Idaho, respectively. The purpose of these studies 
was to alert school personnel to funding changes caused by the 
proposition and the initiative. Finally, Case 15 further 
discussed implications for school funding resulting from the 
passage of Proposition 13 and the One Percent Initiative. 

Initially, an attempt was made to categorize the 15 studies 
according to the four common cost analysis procedures: 
cost-benefit, cost-effectiveness, cost-feasibility, and cost- 
utility. However, it became apparent that the studies fell more 
naturally into the four categories described above. 



In some cases , alternative cost methods could have been used 
that would have changed the category schema used. For example, 
the first three cases could have been considered cost feasibility 
studies if an upper cost limit had been established beyond which 
the programs would no lcnger be considered feasible. The same 
three cases could also have been conceptualized as cost 
effectiveness studies. In assessing the costs of the alternative 
transportation systems , it had been assumed that each alternative 
would provide equivalent services, i.e., would equally well 
transport students to and from school. Given this assumption, 
one could argue that a cost-effectiveness ratio was reported when 
costs were tabulated because the denominator was equal in all 
cases. 

Other examples of possible under-utilization of existing data 
included two cases which simply used archival cost records to 
describe program costa. The same archival records likely 
contained measures of program effectiveness that could have been 
examined in relation to the program costs. Both of these 
programs were widely implemented, federally funded, educational 
programs which had hajd tip be continually accountable for funding 
purposes. In addition, the adopter's guides of Case 8 described 
how to raea3ure program effects, as well as how to estimate costs, 
and could have readily linked the measures of effectiveness to 
the cost estimates, but no such cost-effectiveness linkage was 
found in any of the guides reviewed* 

This review of 15 cost analysis investigations uncovered no 
cost-effectiveness studies. Rather, only simple cost 
descriptions or cost comparisons were conducted. Why were simple 
cost questions consistently posed in lieu of more elegant (e.g. , 
effectiveness or benefit) cost questions, especially when, in 
many cases, existing data could have been analyzed using a more 
powerful cost method? To answer this question, we explored the 
formulation of the otiginal cost question posed by the educator 
or client in these studies. The initial client question largely 
determined the selection of the cost analysis method. 
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After discussions with the principal investigators and 
further review of the cases, we identified four factors which 
contributed to the formulation of the cost question posed by the 
educational client. 

1. The client was not familiar with cost analysis methods: 
In some cases, the clients seemed unaware of the full 
range of possible cost methods and of the related 
questions that could be asked. Consequently, only the 
lowest level, most intuitively obvious, cost questions 
were verbalized (e.g., "What does -the program cost?" 
"Does Program A cost more than Program B?"). 

2. The client did not want to know the relationship of 
costs to effects. Sometimes the client wanted only cost 
description information and had neither the time, 
interest, nor resources to pursue the more complex 
questions. 

3. The client did not consider it possible to collect the 
needed effects data. For example, the client thought it 
was not politically, economically, or logistically 
feasible to collect data, and the data were not already 
available. 

4. The client did not operate in terms of comparative, 
cost-effocts decisions; The "treatment research" logic 
of alternative competing programs did not match the 
"mi -course correction of a single program" reality of 
the client who was more intent on managing than 
researching the educational program. 

The more complicated cost analyses were not conducted because 
the clients often did not know how to ask such cost questions, 
felt no need for information about the relationship of program 
effects and program coste, did not feel it was possible to 
conduct cost-effectiveness or cost-benefit studies, or just did 
not view his/her setting in terms of comparative, cost-effects 
relationships. 

Next, in order to test the validity of our study results and 
interpretations,' we held a conference of 22 evaluators and 
educators f rom tlSD firms, universities, lc al school districts, 
and state departments of education. Four invited speakers 
(representatives of an R&O firm, a state department of education, 
and two school districts) critiqued the study report and 
discussed its conclusions. The conference focused on the issue 
of whether cost analysis is a useful tool in doing educational 
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program evaluations: is it being done? should it be done? can it 
be done? (See Smith 1983b for a copy of the conference 
proceedings.) 

In general , the conclusions of the conference participants 
confirmed and supported our study findings: (1) Although many 
evaluators and educational administrators know little about 
formal cost analysis procedures, there is a need and readiness 
for evaluative cost studies. (2) A few LEAs and SEAs are doing 
such studies, probably more than the journal literature suggests, 
but the study reports are fugitive documents. (3) There are 
serious feasibility problems in using the four formal cost 
analysis methods, and adapted or simplified procedures may be 
needed. (For example, instructional programs are fiscally very 
interconnected and identifying the costs of individual programs 
from current school budget records is problematic—that is why so 
many of the 15 cases we found had to do with support services, 
such as transportation systems, which are budgeted separately.) 
(4) Cost problems usually involve much more than just analysis of 
costs and outcomes, such as the political realities of state 
appropriations and local bond elections. These are the more 
salient dimensions for educational administrators than 
cost-effect ratios. The question is oftentimes not "Which 
program is more cost-effective?" but "What part of the existing 
program can be cut without instructional and political damage?" 

The results of this review of actual studies strengthened cur 
view that cost analysis in education is valued more for its 
possible utility as a management tool than as a "treatment 
research" method as is more prevalent in health evaluations. 
Evidence from the second study suggested that there were probably 
more cost studies being done in education than we had yet 
uncovered because they were management-oriented and consequently 
not published in evaluation journals. Further, there appeared to 
be serious impediments to the implementation of the formal cost 
procedures in education. 

As a result of the first two studies, therefore, we were able 
to conduct a systematic investigation of the use of the various 
forms of cost analysis in education and to identify the major 
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impediments to the implementation of these methods. This work is 
described in Study 3. * 

Study 3: Cost Analysis Practice 
and Impedimenta in SEAs 

In 1983, a wiled questionnaire survey was conducted of all 
50 state departments of education (SEAs) . The purpose of the 
study was to record the types of cost studies SEA evaluation 
units were conducting, the need for future applications of cost 
methods, and existing impediments to the use of cost analysis 
methods. Although almost all SEAs have some personnel with 
evaluation responsibilities, we were interested in the formal 
cost studies conducted by those states with centralized program 
evaluation units. (See Appendix A for copies of the survey 
questionnaire and cover letters.) 

Of the 50 states, only 37 had c j ralized evaluation units in 
1983 and after an initial mailii^ and three follow-up mailings, 
29 of them (78%) provided data for this study. Non-respondent 
bias checks were made using state population data, school 
enrollment figures,' and by direct phone interview with SEA 
personnel. The non-responding states tend to have lower state 
populations and school enrollments than the responding states and 
are more similar to the states without evaluation units than to 
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the responding states. The data reported here are, therefore, 
slightly biased in favor of the more populous states which have 
centralized evaluation units. 

Descriptive information on these evaluation units revealed a 
range of from 1 to 90 inhouse evaluations conducted a year by 1 
to 37 full-time professional staff. The average unit conducted 
about 8 evaluations a year with a professional staff of 7. These 
data were compared with similar information collected five years 
prior. In brief, there appears to have been substantial 
reductions over the five-year period in the size of SEA ■ 
evaluation unit staffs. While the majority of studies continue 
to be conducted inhouse rather than contracted out to external 
consultants, far fewer evaluations are being performed. Although 
the variability across units in terms of size of staff and number 
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of evaluations conducted seems to be decreasing , the individual 
units still respond to a diverse set of multiple 
responsibilities, from consultation to evaluation monitoring to 
policy analysis. (See Smith, 1984, for more information.) 

The results- of the mail survey are presented below. We first 
discuss the current level of practice, then the anticipated need 
for future studies, arr\ finally, impediments to the use of cost 
methods. >— i 

First, we asked the evaluation units to report the number of 
cost studies they had performed in the past five years for each 
of seven different purposes. The list of purposes was developed 
based on our past studies of practice and the literature and are 
presented in Table ) in increasing order of complexity along with 
their reported levels of use. Single cost descriptions and 
cost-feasibility analyses were the most commonly done studies. 
Although a few evaluation units had done as many as a dozen 
studies in the past five years, .he mode for each category was 
clearly zero. 

When asked to describe the most important cost studies they 
had conducted in the last five years, the respondents reported on 
24 different studies. The topics of these studies were evenly 
divided between educational programs (msfch, reading, vocational 
education, compensatory education, ccmmunity education, etc.) and 
support services (data processing, *edia, personnel, assessment, 
etc.). The most used methods were again single cost descriptions 
followed by cost feasibility analyses. In 20 of the cases, the 
studies resulted in important feedback or program changes. Of 
the 15 respondents commenting on the value of their cost studies, 
13 thought they had been valu^e^and effective. Thus, although 
the incidence of cost studies has been low and the methods used 
have been relatively simple, the evaluation units doing these 
studies have generally been pleased with their efforts. 

When asked if there is currently a formal expectation or 
requirement within the state agency that the evaluation unit do 
some form of oost analysia work, a surprising 48.3 percent said 
yes. While 11 of the units (currently devote no resources to cost 
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Table 1 

Cost Studies Conducted by SEA Evaluation 
Units in Past Five Years 



Purpose of Study Number of Studies 

Mean Range 

Single Cost Description 
To describe the costs of a single existing 1.80 0-15 

program more fully ,. 

Coot-Feasibility Analysis 
To see if a planned program is affordable 1.47 0-15 

within the resources available 

Cost-Utility Analysis 
To compare the costs of a planned program 0.59 0-10 

with its estimated outcomes 

Single Coat-Outcome Description 
To compare the costs with outcomes of a 0.<93 0-6 

single existing program 

Multiple Coat Descriptions 
To compare the costs of 2 or more existing 0.14 0-1 

programs 

Cost-Effectiveness Analysis 
To. compare the costs with outcomes of 2 or 0.35 0-5 

mor e existing programs * where the outcomes are 
not measured in dollars but in test scores, 
ratings^ etc. 

Cost-Benefit Analysis 

To compare the costs with outcomes of 2 or 0.31 0-5 

more existing programs , where both outcomes 
and costs are measured in dollars 



studies, the other 17 units responding devote an average of 11.5 
percent of their budgets to cost work. When asked if they 
anticipate that over the next five years there will be a formal 
expectation or requirement for cost analysis work, 58. 6 percent 
said yes and anticipated that they would have to spend an average 
of 17 percent of their budget on cost studies. Therefore, more 
SEA evaluation units expect to be required to do cost studies in 
the future and anticipate having to spend a greater share of 
their resources on cost work. 
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Finally, we asked the evaluation units how many of the seven 
types of cost studies listed earlier (see Table 1) they 
anticipated doing over the next five years. Figure 1 contains a 
comparison of current to anticipated use of the seven types of 
cost studies. Evaluation personnel expect to do more of all but 
the simplest procedure (single cost description) , with the 
greatest increases being in the use of cost utility analysis 
(18.3% increase) and cost feasibility analysis (14.8% increase). 
Neither of these latter two approaches include the direct 
assessment of outcomes as with the cost-effectiveness method, for 
which the expected increase was more modest (about 11%). Thus, 
while there may be an increased demand for cost analysis and 
greater evaluation resources devoted to such studies, the 
complexity of the methods used may not increase appreciably. 

Even though almost 60 percent of the SEA evaluation units 
anticipate doing cost studies in the future, they expect to be 
using the ostensible "cost method of choice," cos^-effect iv«i»ess 
analysis, only about a quarter of the time. What impediments are 
there to doing cost studies in these settings and especially to 
the use of cost-effectiveness analysis? We next addressed those 
issues. 

Through studying the literature (Study 1) , examining examples 
of cost analysis service work (Study 2) , and discussing the 
problems of using cost analysis methods with evaluators, clients, 
and economists, we had developed a preliminary list of 
impediments to the use of cost methods. Our original list 
contained 51 total items— 21 items in the category of resource 
and organizational constraints and 30 items in the category of 
methodological and technical inadequacies. By editing similar 
items, regrouping others, and eliminating the more esoteric ones, 
we reduced our list to 10 under each of the two categories and 
then, after field tests, to a final 14 items. Responses to this 
list of impediments are rummarized in Table 2. 
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Figure 1. Percent of SEA Evaluation Units Using Cost Analysis Methods 



Table 2 

Impediments to the Use of Cost Analysis in 
SEA Evaluation Units 



Item Mean * 

a. We are seldom asked to do cost studies 3.00 

b. Decision makers are not often interested in 

actual cost information 2.28 

c. Decision makers do not often use the results 

of cost studies 2.30 

d. It takes too much time to conduct cost studies 2.37 

e. It costs too much to conduct cost studies 2.22 

f. Accurate cost data are usually not available 2.71 

g. Cost study results are incomplete because it 
is not possible to include all important 

cost factors 2.69 

h. Accurate outcome data are usually not available 2.69 

i. it is difficult to relate cost data to 

educational outcomes 3.00 

j. Cost results do not tell managers how to 

improve program operations 2.43 

k. We lack staff with the technical capability 

to conduct cost studies 2.45 

1. We lar!; a 'ailable consultants or experts to 

help us conuuct cost studies 2.31 

m. We do not have sufficient experience in 

conducting cost studies 2.66 

n. We have few guidebooks, texts, or examples 

to follow in conducting cost studies 2,82 

* 4 - Strongly Agree 
3 - Agree 
2 * Disagree 
1 ■ Strongly Disagree 

Mean is based on N of 29 minus number of Don't Knows 
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The three strongest impediments to doing colpt analysis 

were T 

(a) We are seldom asked to do cost studies] 

(i) It is difficult to relate cost data to Educational 

outcomes. / 1 

(n) We have few guidebooks, texts, or examples to 

follow in conducting cost studies. 

Respondents were least able to decide (i.e., marked "Don't 
Know") whether it was possible to include all important cost 
factors in an analysis (item (g)) and whether consultants or 
experts were available to help with cost problems^ (item (1)). A 
reliability coefficient (standardized item Alpha) was computed 
for this scale using only the observ tions on which complete data 
were available (ft*20) and was found to be .64. The most highly 
intercorrelatep items were studies costing too much and taking 
too much time/ ((d) with (e) - .72)? decision makers not 
interested And not using cost results ((b) with (c) ■ .70); and 
decision makers not interested and not asking for cost studies 
((a) with (b) - .66). 

Because cost-effectiveness analysis is often advocated as the 
best method for use in educational evaluation but is seldom 
employed, we asked a similar question about the impediments 
specific to its use. Responses to that question are summarized 
in Table 3. The respondents were less critical of any individual 
impediment than in the previous question. The greatest 
impediments to uring cost-effectiveness analysis were 

(g) Cost effectiveness analysis is difficult to do 

because of technical details (e.g., discount rates) 
and the need for sophisticated analysis procedures. 

(e) Cost effectiveness analysis considers only a 

limited number of program outcomes and so does not 
represent true program effects. 

(b) Cost-effectiveness analysis is often not feasible 
because comparative program studies are seldom 
possible. 

A standardized item Alpha coefficient of .73 was found for 
this scale using only responses with complete data (N ■ 17). The 
highest intercorrelations were between cost-effectiveness 
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Table 3 

Impediments to the Uae of Cost-Effectiveness 
Analysis in SEA Evaluation Units 



Item Mean* 

Cost-effectiveness analysis is often unnecessary 
because decision makers are not interested in 
relating program costs* and effects 

Cost-effectiveness analysis is often not 
feasible because comparative program studies 
are seldom possible 

Cost-effectiveness analysis is too specialized 
a technique to be generally applicable r 

Cost-effectiveness analysis is so costly and 
complex that it is warranted only for 
major studies 

Cost-effectiveness analysis considers only a 
limited number of program outcomes and so does 
not represent true program effects 

Cost-effectiveness analysis is of limited 
utility since it provides no information on 
program procedures or local conditions 

Cost-effectiveness analysis is difficult to do 
because of technical details (e.g., discount 
rates) and the need for sophisticated 
analysis procedures 

Cost-effectiveness analysis cannot be applied 
to educational programs without further 
development of the method 

4 - Strongly Agree 
3 * Agree 
2 * Disagree 
1 * Strongly Disagree 

Mean is based on N of 29 minus number of Don't Knows 




2.38 
2.44 

c 

2.08 
2.35 
2.45 
24 24 

2.56 
2.25 



analysis needing more development and its being too technical 
((g) with (h) ■ .74); it needing more development and it 
considering only a limited numbei: of outcomes ((e) with (h) ■ 
. 74) ; and between it being too specialized and often not feasible 
((b) with (c) * .71). o 

The previous responses suggest that there will be an increase 
in the use of cost methods in SEA evaluation work, especially 
with the simpler methods. The major impediments include not 
being asked to do such studies (which appears to be changing) , 
the technical difficulty of relating costs to outcomes, and the 
lack of supporting guidebooks and examples, with the specific 
cost-effectiveness analysis technique per se, the situation is 
somewhat different. There is greater uniformity of response (the 
range of the means for cost-effectiveness analysis spans only .48 
points, see Table 3, as compared with .78 points for cost 
analysis in general, see Table 2) and no particular impediments 
seem to dominate as a major problem. Although fewer evaluation 
units expect to be increasingly using -cost-effectiveness 
analysis, they do not cite major impediments to its use. Why 
might this be so? 

Notice the larger number of "Don't Know" responses for 
cost-effectiveness analysis as compared with the previous 
question (Table 3 versus Table 2). These educational evaluators 
apparently felt they knew less about the specific 
cost-effectiveness analysis technique than they did about cost 
analysis in general. In a separate question, we asked 
respondents to indicate their highest level of familiarity with 
cost-effectiveness analysis in terms of (a) having no knowledge 
of it, (b) having minimal familiarity with it, (c) having read 
about it or studied it, and (d) having conducted studies using 
it. Eighteen (62%) of the respondents reported having studied or 
used the techniques while the remaining 11 (38%) said they knew 
little or nothing about it. Perhaps the data in Table 3 reflect 
two different groups of evaluators: some units know little about 
the technique and so marked "Don't Know," while those familiar 




with the ^technique reported that none of the possible impediments 
were raaj^r obstacles; that is, the means are lower in Table 3 
(2.34 average) than in Table 2 (2.57 average). 

Several other correlations support this possible "two group" 
explanation. There is a significant relationship between the 
total number of cost studies a unit has done in the past five 
years and the number they expect to do over the next five years 
(R - .69, significant at p - .001), and there is an extremely 
strong relationship between those units that have had formal 
expectations of cost work and those that anticipate future 
demands (R - .94, significant at p - .001). Purther, those units 
most expecting to use cost-effectiveness analysis in the future 
are those who have done the most cost work in the past (R » .63, 
significant at p ■ .001) or who have used cost-effectiveness 
procedures most in the past (R ■ .49, significant at p ■ .003). 
Finally, we recoded the data summarized in Tables 2 and 3 to 
compute (1) a general cost analysis impediments index and (2) a 
cost-effectiveness impediments index, respectively. These 
indices, in general, reflect the perceived magnitude of the 
problems that the SEA evaluation units feel they must overcome in 
order to conduct cost studies. There is a moderate, positive 
correlation between these two indices (R ■ .31, significant at 
p » .049), suggesting that those units who feel major impediments 
in doing cost analysis in general also feel impediments in using 
the cost-effectiveness technique. 
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Discussion 

Summary of Results 

In general, we can summarize the major findings of the 
foregoing work as follows. 

There are few cost analysis studies of educational programs 
published in the educational literature. There are many more 
cost studies published in the health literature, especially more 
cost effectiveness studies. The health applications tend to be 
more treatment research oriented than program management oriented 

30 

36 



as in the educational studies. Health researchers have 
apparently solved some of the implementation problems of using 
cost methods , although they often merely assume outcome effects. 
There are reportedly many methodological shortcomings in the cost 
studies health researchers are conducting, but there continues to 
be a need for them and the literature continues to urge their use 

Our study of an R&D firm's use of cost analysis illustrates 
the educational use of the methods for program management more 
than for treatment or intervention comparisons. Only the 
simplest of cost methods were used, perhaps in part due to 
clients (a) not knowing enough to ask for more complex analyses, 
(b) not wanting to relate costs to outcomes, (c) not considering 
complex cost studies feasible, or (d) not thinking in terms of 
comparative trials, but more in terms of mid-course corrections 
of single programs. A conference with educational administrators 
and evaluators emphasized the need for cost methods in 
educational evaluation and suggested that there may' be a greater 
number of fugitive management uses of cost analysis than we had 
yet uncovered. Major feasibility problems were also noted. 

Our study of SEA evaluation units confirmed that some units 
are doing cost work and that many of them are under formal 
obligations to do so. They expect these obligations to increase 
and anticipate that they will be doing more studies in the 
future. These units expect to be using the slightly more complex 
methods in the future, but do not anticipate extensive use of the 
theoretical "method of choice," cost-effectiveness analysis. 

The biggest impediments to doing more cost work in SEA 
evaluation units, aside from not being asked to, is the 
difficulty of relating costs to outcomes and the lack of texts 
and examples to guide their work. The major impediment to the 
use of cost-effectiveness analysis is its complexity, although 
many of the respondents seem to lack sufficient knowledge of this 
technique to comment on impediments to its use. 
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A Knowledge Transfer Interpretation 

In closing, we would like to comment briefly on the 
possibility of using knowledge transfer research and principles 
to explain further or illuminate our findings. Of the four major 
•knowledge transfer dimensions (organizational characteristics, 
innovation characteristics, individuals, and innbvation process), 
we have only a little information on the first three. We can 
make only loose inferences, since our work was not designed as a 
knowledge transfer study, and the impediments we employed evolved 
from our previous studies rather than being theoretically 
derived. A look at knowledge transfer principles might be 
instructive, however, in understanding why cost methods are not 
more widely used in education. 

There are a number of knowledge transfer models that might be 
used in interpreting our work. Stenzel (1982), for example, has 
described 8 person and 18 institutional barriers to implementing 
new evaluation methods in state departments of education. These 
barriers represent the problems in adapting new Methods as seen 
from an insider's point of, view, or what Ross (1974) calls the 
"buyer's" perspective as contrasted with the innovator's or 
"seller's" perspective. Stenzel' 8 list as well as other 
collections of innovation factors, forces, conditions, and 
attributes (e.g., Davis and Salasin, 1979; Zaltman et al. , 1977; 
Havelock, 1974— see Glaser et al. , 1983, for a general summary of 
this literature) covers a wide range of dimensions on which we do 
not have comparable information across the sites in our study. 
Since most of the impediments in our study concern the nature of 
the innovation itself (approximately 7 of the 14 impediments for 
cost analysis in general - Table 2, and 7 of the 8 impediments 
for cost-effectiveness analysis - Table 3), we have chosen 
Glaser's (1973) CORRECT summary of innovation attributes for use 
here. 

Glaser (1973) discusses seven attributes of an innovation 
that facilitate its transfer: credibility; observability; 
relevance; relative advantage; ease in understanding and 
installation; compatibility; and trialability , divisibility or 
reversibility. Each of **.hese attributes is discussed below with 
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reject to the evaluation use of cost analysis methods. Only 
th^>se possible impediments with a score of 2.50 or higher! 
(indicating that the item is an actual implementation bartier) 
are used in the discussion which follows. 

I Credibility of the innovation refers to the evidence of its 
value from past use or from support by respected persons. There 
/ does not seem to be a problem with the credibixity of these cost 
methods given the widespread formal requirements for their use in 
SEA evaluation work. Both theorists and practitioners continue 
to agree on t,he need for these methods in addition to the evident 
administrative support. \; 

Observability refers to the opportunity for potential users 
to see demonstrations of the operation or results of the 
innovation. This may be a problem with the cost analysis methods 
in that the SEA respondents reported lacking examples to follow 
in conducting studies (item n. , Table 2). 

Relevance concerns the promise of the innovation in coping 
with a serious and persistent problem. As with the credibility 
criterion, cost methods appear to be viewed very positively in 
this regard— they are continually advocated in the literature as 
a solution to educational cost problems. 

Relative advantage of the innovation refers to the extent to 
which the resulting improvement will be worth the investment 
required and will result in better outcomes than current 
methods, it appears that cost methods in general are seen as 
having relative advantage over non-cost analysis approaches. 
However, the anticipated increase in the use of the simpler cost 
methods instead of the more complex approaches (see Figure 1) 
suggests that such methods as cost-effectiveness and cost-benefit 
analysis are not seen as having a relative advantage over, say, 
the simpler cost feasibility and cost utility analyses. 

It is with the ease of understanding and installation 
criterion that transfer of the cost methods seems to have the \ 
greatest difficulties. Five of the seven significant impediments^ 
in Table 2 refer to installation problems: cost data not 
available (item f. ), outcome data not available (item h. ), 
inability to include all important cost factors (item g.), 
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difficulty in relating cost to outcome data (item i.)# and lack 
_o£ guidebooks and examples t6 follow (item n.). The only 
significant cost-effectiveness impediment (item g. , Table 3) also 
fell in this category — the complexity of the method. 

Respondents also reported not having sufficient experience in 
conducting cost studies (item m. , Table 2). This barrier could 
be seen as another installation problem or as a lack of 
compatibility , which reffers to the match between the innovation 
and the potential users 1 previous values and procedures, 

Trialability , divisibility # or reversibility refers to the 
ability to use the innovation without an irreversible commitment 
of the organization. Cost methods seem to offer little 
difficulty in this area. « / 

In summary, the use of Glasser's (1973) CORRECT framework 
emphasizes that the major problems in the use of cost methods in 
SEA evaluation work have to do with the difficulty of , 
understanding and implementing the method. The use of cost 
analysis as an innovative procedure scores high on other criteria 
such as need, credibility, and relevance. Some SEA units simply 
are not requested to do such studies (item a. , Table 2), but 
those who are attempting to use cost methods are having the 
greatest difficulty with the actual implementation and use of the 
procedures. Attempts to assist in the adoption of this , 
innovation should probably focus, therefore, on the development 
of guidebooks, texts, and examples fot practitioners' use. The 
activities most likely to support the use of cost methods in 
educational evaluation at this time would seem to be materials 

development, training, and technical assistance. 

/ 
/ 

/ 

i / 
/ 

/ 
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Copy of first survey cover letter was lost, but the contents 
of the letter were similar to that of the second follow-up 
letter on page 47. 
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Postcard: First Follow-Up Notice 



August 30, 1983 

Laat wNk wa aant you a quaationnaira on tha uaa of coat analyaia aathoda 
in your evaluation unit and on budgating for avaluatlon atudlaa. 

If you hava alraady sant it baok, plaaaa aooapt our alnoara thank a. Youc 
contribution to thla atudy la aoat appraolatad. It you hava not yat 
returnad it, plaaaa do ao today. laoauaa tha quaationnaira waa aant to 
only ana parson in your atata departaant, it ia asaantlal that wa hava 
your quaationnaira if wa ara to undaratand what raally happana in aaoh 
atata sduoatlon ngancy. 

If by aoaa ohanoa you did not raealva tha quaationnaira, or it got 
alaplacad, plaaaa call ua right now, on our toll- era a nuabar 
1-800-547-6339, and wa will gat anothar ona in tha sail to you today. 



Sincaraly, 



Jana Kay Smith. Ph.D. 
Raaaaroh on Evaluation Program 



Nick L. Saith, Ph.D. 

Raaaarch on Evaluation Program 
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Second Follow-Up Notice 



Northwest 

Educational 
Laboratory < 



300 8.W. Sixth Avwhm • Portland, Oregon 97204 
T«Uphon«: (603) 248-4600 TELEX: 701716 CABLE: NWREL 



September 15 , 1983 



Dear 



About three weeks ago we wrote to you asking about evaluation budgeting 
and the use of cost analysis methods in your state department evaluation 
unit. As of yet, we have not received your completed questionnaire. 

We are gathering this information in order to help state department of 
evaluation units like yours respond to the increased need for cost 
analysis information within evaluation work. 

We are writing to you again because of the importance of including your 
questionnaire in our study. Because the questionnaire was sent: to only 
one person in your state department, it is essential that you return your 
completed copy if we are to understand what really happens in each state 
evaluation unit. 

In the event that your questionnaire has been misplaced, we have enclosed 
another copy with a stamped, self-addressed return envelope, if you have 
any questions or comments, please don't hesitate to call our toll-free 
number 1-800-547-6339. 

Your cooperation is greatly appreciated. 

Cordially, 





Visiting Scholar 

Research on Evaluation Program 



Nick L. Smith, Ph.D. 
Director, 

Research on Evaluation Program 



eg 

Enclosures 



An EQUAL OPPORTUNITY EMPLOYER 
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Thank You Letter 



Northwwt 
Regional 
Educational 
Laboratory < 




300 8.W. Sixth Avonuo • Portland, Oregon 97204 
TELEX: 701716 CABLE: NWREL SOURCE: STL058 



(503) 248-6800 



September 17, 1984 



Dear 



Several months ago you assisted us with a survey about the use of cost 
analysis methods by evaluation units in state departments of education. 
He are very grateful for the help you provided us and are writing to 
share with you some of the results of the survey. 

Of the 50 state departments of education, only 37 had centralized 
evaluation units in 1983 and 29 of them (78%) responded to our 
questionnaire. Although a few evaluation units had done as many as a 
dozen ccst studies in the past five years, the mode was clearly zero. 
Single program cost descriptions and cost-feasibility analyses were the 
most commonly done studies. Although the number of studies done was low 
and the methods used were relatively simple, the units doing cost studies 
reported being pleased with the impact of their efforts. 

Surprisingly, 48 percent of the units said there was currently a formal 
expectation or requirement that they do some form of cost analysis work 
and they estimated devoting an average of 11.5 percent of their budgets 
to cost work. Pifty-nine percent of the units anticipated having a 
formal requirement to do cost studies in the next five years and expected 
to spend an average of 17 percent of their budgets for that purpose. 

When we asked why the units were not currently doing more cost studies, 
some respondents said that they were simply not being asked to do them. 
Most respondents, however, said that it was difficult to relate cost data 
to educational outcomes and that they had few examples, texts, or 
guidebooks to follow in conducting cost studies. 

The evaluation units did expect to be using a wider range of cost methods 
in the future, especially cost-utility analysis and cost-feasibility 
analysis. Few of them expected to be using the more complicated methods 
which include actual comparisons of costs with outcomes, such as 
cost-effectiveness and cost-benefit analysis. 

When we asked why they didn't use cost-effectiveness analysis more (the 
theoretically best method for this work) , they reported either not 



knowing much about it or thought 
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AN EQUAL OPPORTUNITY EMPLOYER 



Page 2 

September 17, 1984 

w 

* 

We are continuing our work on cost analysis methods in 1985, including 
developing more streamlined methods for evaluation use and producing 
guidebooks and examples to help evaluators get started with the methods. 
We welcome your suggestions, questions, and requests for materials. 

Enclosed is a copy of Henry Levin's new book Cost-Effectiveness: A 
Primer which was developed in part with our support. We would also be 
happy to send you a copy of the full survey report "Cost Analysis in 
Educational Evaluation 11 upon request. Thank you again for participating 
in our survey study. 

Cordially, 

Nick Le Smith, Ph.De Jana Kay Smith, Ph.D. 

Research on Evaluation Prog reus 

NLSseg 
Enclosure 
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EVALUATION BUDGETING AND COST ANALYSIS 

I 




This Is a survty of state education agency evaluation units to (1) look at the ways 
evaluation studies are budgeted, (2) find out how often cost analysis studies are 
conducted, and (3) Identify possible problems In doing cost studies. 

Please answer all of the questions. If you wish to content on any questions or qualify 
your answers, please use the Margins or a separate sheet of paper. 



Conducted by: Research on Evaluation Program 

Northwest Regional Educational 
300 S. U. Sixth Avenue 
Portland, Oregon 97204 
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QUESTIONS ON EVALUATION MEETING 

In order to usist evcluators In Managing resources, wt art studying how funds art 
typically allocated across cost categories within an evaluation study. He would tike to 
know, thtrtfore, how your unit would typically budget "evaluation funds for two types of 
comma evaluation studies. 

THE SITUATION: Questions 1 and 2 refer to the following hypothetical situation. 



Assum that your evaluation unit has been asked tc evaluate a Minority education project 
1n a nearby Metropolitan city In your state. The project Is designed to iMprovt student 
self toage and school ptrforMance, and to Increase the conaunlty acceptance of Minority 
students. In this first year of the project, your unit Is to conduct a fonaatlve 
evaluation to provide feedback for progriM Improvement to the local school district. You 
have $10.000 total to spend on all expenses of tht evaluation study. We would like to 
know what Major evaluation activities you would Include 1n your study (Question 1) and 
how you would allocate the evaluation resources (Question 2). 



Q-l EVALUATION DESIGN; 



CHECK ALL ACTIVITIES THAT WOULD BE A WJOR PART OF YOUR F ORMATIVE 



a. Scvolopaient of an evaluation 
plan ° 

b. Use of an evaluation 
advisory panel 

c. DevtlopMent and piloting of 
InstntMents 

d. Use of control or coMparlson 
groups 

ft. Collection of pre- test data 

f. Review and analysis of 
docuMants 

g. Collection of on-site 
observation data 

h. Collection of personal 
Interview data 



j. Collection of Mall survey data 

k. Use of coMMunlty hearings 

. 1. Use of expert reviews 

mmm m. Collection of post* test Cat a 

_ n. Development of case study 

reports 

o. Statistical analysis of data 

p. Verbal presentation of study 

findings 

^ q. DevtlopMent of narrative 
reports 

tmmmmm r. Provision of consultation and 
followup assistance to project 
staff 



1. Use of phone Interviews 
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Q " 2 Sftiifli^lWgfl^gi^^S, Sltgff m *J™1 Y0U ™ 10 LOCATE TO EACH 
C ?!!?r ^.I-'Lx^Ii^, EVW.UATIOH (remember the total amount you have to 
allocate Is SlO.OOOl 

. Parsonnel costs (proftsslonal staff and support staff salaries 

and benefits) 

-, °. Consultant costs (consultant salaries and benefits) 

- c. Staff travel 

j d. Consultant travel 

-» o. Cowaunlcatlon (telephone, postage, shipping) 

f. Fac111t1es(f acuities «d utilities) 

g. Duplication (printing and duplication) 

. h. Data processing 

Supplies (Materials, supplies, equipment) 

J. Other services (e.g., subcontracts, conferences; please specify) 



010.000 TOTAL COSTS 




O 
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THE SITUATION: Questions 3 and 4 rtftr to tht following hypothetical situation. 



Assume no* that It Is tht third year of this demonstration Minority education project and 
that your unit has been asked to conduct a simmtlvt evaluation to de tannine triiether the 
project should be adopted throughout the state. You again have $10,000 total to spend on 
all expenses of tht evaluation study. Please Indicate what activities you would Include In 
the study (Question 3) and how you would allocate your resources (Question 4). 



Q-3 EVALUATION OESISW : CHECK ALL ELEMENTS THAT MOULD BE A MAJOR PART OF YOUR SUWATIVE 
EVALUATION STUDY • 



«• Developmnt of an evaluation 

plan 

b. Use of an evaluation 

advisory panel 

_ c. Development and piloting of 
" instruments 

d. Use of control or comparison 

groups 

m e. Collection of prt-ttst data 

f. Review and analysis of 

d o c u ments 

, g. Collection of on-site 
observation data 

h. Collection of personal 

Interview date 

1. Use of phone Interviews 



J. Collection of Mil survey data 

k. Use of community hearings 

1. Use of expert reviews • 

■ m. Collection of post- test } data 

n. Development of cast study 
reports 

o. Statistical analysis of data 

p. Verbal presentation of study 

' findings 

, q. Development of narrative 
reports 

. r. Provision of consultation and 
followup assistance to project 
staff 
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<M EVALUATION BUDGETING: INDICATE THE AMOUNT OF MONEY YOU WOULD ALLOCATE TO EACH CATEGORY 
FOR TMS syMMArfvTfYALUATION (remember the total wount you have to allocate 1s 

»• t**"*?^!?*? eo,ts (P™'*" 1 ™*! »taff and support staff salarlts and 

bono' its i * 



b. Consultant costs (consultant salaries and benefits) 

c. Staff travol 

d. Consultant travol 

0. Communication (telephone, postage, shipping) 

f. Fac111t1es(fac111t1es and utilities) 

g. Duplication (printing and duplication) 

h. Data processing 

1. Supplies (Materials/ supplies, equipment) 

J. Other services (e.g., subcontracts, conferences; please specify) 



#10,000 TOTAL COSTS 



Q-5 Finally, with respect to both of these situations, what Indirect or overhead rate 
would your agency normally add to the costs of these evaluations? For example* a 20* 

2 v ! rh i! d r !i* wou1 l br !2! *"! toUl »«* of "» of these studies to $12,000. (20*"T 
$10,000 - $2,000; $10,000 ♦ $2,000 • U2.000) 

WHAT OVERHEAO RATE WOULD YOUR AGENCY USE FOR THESE STUDIES? ^PERCENT 

Thank you for answering these questions on evaluation budgeting. The next section of the 
questionnaire pertains to using cost analysis methods In evaluating educational programs. 
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QUESTIONS ON COST ANALYSIS STUDIES ANO HITHOOS 

Some stitt department of education evaluation units receive requests for cost 
information. In ordar to assist them, we mod a more complete understanding of what 
kinds of cost work aro currant 1y bolng conducted, what work Is likely to bo requested, 
and antlclpatad problems In doing cost work, m aro ssptelally Intorostad In formal 
studios Involving tha computation of program costs. 

Q-« First wa ntad to know what kind of cost stuJIes, If any, aro balng dono. Listed 
below are several purposes that cost analysis studies, can serve. Please write the 
number of cost studies conducted for each purpose by your evaluation unit In the last 
five (S) years and the approximate nueber anticipated over the next five (S) years. 

NUMBER OF COST STUDIES 

PURPOSE OF COST STUDY: DONE IN PAST ANTICIPATED 

S YEARS OVER THE 

NEXT S YEARS 

a. To see If a planned program Is affordable within • 

the resources available (cost-feasibility analysis) 

b. To compare tha costs of a planned orooreo with 

Its estimated outcomes (cost-utility analysis) 



c. To describe the costs of a single existing 
program sore fully (single cost description) 

d. To compare the* costs with outcomes of a single 
existing program (single cost-outcome description) 



e. To compare the costs of 2 or more existing 
programs (multiple cost descriptions) 

f . To compare tha costs with outcomes of 2 or more 
existing programs, where both outcomes and costs 
are measured in dollars (cost-benefit analysis) 

g. To compare the costs with outcomes of 2 or more 
existing programs , where the outcomes are not 
Measured in dollars but In test scores, ratings, 
etc. (cost-effectiveness analysis) 



h. Other (please specify) 



Q-7 Is there currtntly i formal expectation or requirement within the agency that your 
unit do some for* of cost analysis work? 

NO YES PLEASE EXPLAIN: . 



Approximately what percent c" your total evaluation 
unit resources are currently devoted to performing 
sow form of cost analysis work? 

PERCENT 



Q~8 Do you anticipate that over the next 5 years there will be a formal expectation or 
requirement that your unit do some form of cost analysis work? 

NO YES PLEASE EXPLAIN: ; 



Approximately what percent of your total evaluation 
unit resources do you believe will be devoted to 
performing some form of cost analysis work over the 
next 5 years? 

PERCENT 



If you have conducted no cost studies In the last 5 years, skip to QUESTION 10* 
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Q-9 Please briefly describe the two Most Important cost studios you conducted In tht last 
S years. Mease Indicate: 

Cost Study I 

a. Tht topic (e.g., bus loosing, a rtading program): 

b. Tyf . of cost study (soo Question 6, a-h for typology): 

c. Basic procedures: 

d. The decision or actions resulting from the study: 

e. Your assessed* of the value of this particular study: 



Cost Study 2 

a. The topic (e.g., bus leasing, a reading program): 

b. Type of cost study (see Question 6, a-h for typology): 

c. Basic procedures: 

d. The decision or actions resulting from the study: 

e. Your assessnent of the value of this particular study: 
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Q-10 Wt recognize that some (valuation units way have had Uttlt or no experience 

conducting cost studios. Evtn If this Is the cast with your unit, your responses to 
this survey are Important to us. We are Interested in the problems evaluitlon units 
are having 1n conducting cost studies, end especially why they do not conduct more 
of these studies. 

• 

Some units report a number of Impediments that prevent than from doing cost 
studies. Please Indicate the degree to which the following fictors restrain the 
number of cost studies done by your evaluation unit. 

THESE FACTORS IMPEDE OUR DOING COST STUDIES; £ A, 

(circle degree of agreement or disagreement for & Jr £j? £ 

each factor) $f $ <f <*f <f 

a. We art seldom asked to do cost studios SA A D SO OK 

b. Decision makers are not often Interested 1n actual cost 

Information . . . SA A D SO DK 

c. Decision makers do not often use the results of cost 

Studies SA A 0 SO OK 

d. It takes too much time to conduct cost studies SA A 0 SO OK 

e. It costs too much to conduct cost studies SA A 0 SO DK 

ft Accurate cost data art usually not available SA A D SO OK 

g. Cost study results are Incomplete because 1t 1s not 

possible to Include all Important cost factors SA A 0 SO DK 

h. Accurate outcome data are usually not available SA A D SO OK 

1. It Is difficult to relate cost data to educational 

outcomes SA A 0 SO DK 

J. Cost results do not tell managers how to Improve program 

operations SA A D SO DK 

k. We lack staff with the technical capability to conduct 

cost studies SA A D SO OK 

1. We lack available consultants or experts to help us 

conduct cost studies •••••• SA A D SO DK 

m. We do not havt sufficient experience 1n conducting cost 

studies . . . . SA A D SO OK 

n. He have few guidebooks, texts, or examples to follow In 

conducting co*t studies . SA A 0 SO OK 

o. Other (please specify) SA A 0 SO OK 
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Q-U One type of cost analysis proctduri, cost'affoctlvonoss analysis , comparts all tha 
costs (e.g., personnel, facilities, equipment) of two or work similar programs to 
maasurts of program outcomas (t.g., tast scores, behavioral changes). 

a 

CHECK YOUR HI6HCST LEVEL OF FAMILIARITY WITH COSMFFECTIVEftESS ANALYSIS (chtck one); 

Have conducted studies using It 

Have studied or read about It 

Have minimal familiarity with It 

____ Have no knowledge of It 

Q-12 Cost-effectiveness analysis Is one of the most suitable way, of comparing the costs 
and outcomes of one education progras with another, but so far the technique has 
been used little In educational evaluation. To understand why this technique Is not 
widely used In educational evaluation, wa would like your opinions about the 
following Items. 

00 YOU THINK THAT: 

(circle degree of agreement or disagreement for *• . & 

At//// 

i. Cost-effectiveness analysis Is often unnecessary * T 0 ~ 

because decision natters are not Interested In 

relating program costs and effects SA A 0 SO OK 

ba Cost-effectiveness analysis Is often not feasible 

because comparative program studies are seldom i jsslble. SA A 0 SO OK 

c. Cost-effectiveness analysis Is too specialised a 

technique to be generally applicable SA A 0 SO OK 

da Cost-effectiveness analysis Is so costly and complex 

that It Is warranted only for major studies SA A 0 SO OK 

e. Cost-effectiveness analysis considers only a limited 
number of program outcomes and so does not represent 

true progrma effects SA A 0 SO OK 

fa Cost-effectiveness analysis Is of limited utility 
since It provides no information on progrma 

procedures or local conditions ; . SA A 0 SO OK 

g. Cost-effectiveness analysis Is difficult to do because 
of technical details (e.g.. discount rates) and the 

need for sophisticated analysis procedures SA A 0 SO OK 

h. Cost-effectiveness analysis cannot belpplled to 
educational programs without further development of 

the method ' SA A 0 SO OK 

Thank you for answer 1 in these questions on cost analysis! The next section of the survey 
pertains to the operation of your evaluation unit. 
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QUESTIONS ON YOUR EVALUATION UNIT 



In order to understand the differences anong state dcparteent evaluation units with 
respect to evalu 
characteristics 



respect to evaluation budgeting and cost analysis, we need to know a few basic 
of your operation. 



Q-13 Mhat was the total annual budget for your evaluation unit for the 1982-1983 school 
year? 

TOTAL BUDGET: S 



Q-14 How many full-tie* equivalent (FTE) professional staff worked In your unit during 
the 1982-1983 school year? 

TOTAL STAFF: FTE 

Q-15 Som state departeent evaluation units conduct Most of their evaluation studies by 
using Inhouse staff, other units by subcontracting to external evaluators. During 
the 1982-1983 school year, how Many of your evaluation studies were done by your 
staff, how Many through subcontracts? 

NUMBER INHOUSE EVALUATIONS: 

NUMBER EXTERNAL EVALUATIONS: 
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Q-16 Many tviliMtlon units have responsibilities othtr than conducting evaluation 

studios. Please Indicate below the eajor responsibilities of your evaluation unit 
during 1962-1983. 

CHECK ALL MAJOR RESPONSIIILITIES: 

a. Evaluation studies 

' b. Evaluation Monitoring 
c. Testing program 

o\ 'laming 

e. Research studies 

____ f. Consultation/technical assistance 

_ g. Policy analysis 

h. Reeds assessment 

_ 1. Infcmatlon provision 

j J. Other (please specify) . 



Finally, we would like to ask a few questions so that we cat better support state 
department evaluation units In 1984 and '1985 In Improving tlelr evaluation efforts. 

Q-17 What parts of your total evaluation process most need Improvement over the next 3 to 5 
years If you are to provide the best possible service t 

CHECK ALL MAJOR AREAS NEEDING IMPROVEHENT: 

a. Problae Font at Ion 

b. Study Design 

c. Data Collection 

d. Data Base Maintenance 

e. Data Analysis 

f. Information Reporting 

g. Evaluation Management 

h. Other (plaase specify) 
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Q-18 What sptclflc problam do you antic oatt facing with n tht ntxt 3 to 5 ytirs that 
will afftct tht quality of tht strvlcts you provldt (t.g., H«1ttd funds; untrslntd 
or foMtr staff; Incrtattd dtaands for tvaluatlon strvlcts; lack of sultablt 
tvaluatlon Mthods)? 



1. 



2. 



4. 



Q-19 If statt or ftdtral funds *vrt tvallablt for rtstarch and davalopotnt *ork to 
1*rovt statt ltvtl tvaluatlon, what topics or actlvltlts should rtcalva tost 
atttntlon? 



1. 



2. 



3. 



4. 
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Some state departments art struggling with traditional testing problems, while others are 
feeing new difficulties brought about by the widespread use of Microcomputers. 

Q-20 Mow do you think statewide assessment progreas can be Improved so that test results 
are acre useful to local districts? 



Q-21 Is the increased availability of Microcomputers likely to significantly affect the 
nature of your evaluation work? 

NO YES PLEASE EXPLAIN: ' 
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Think you for your fntorost and support in providing us with this Information. Mc vary 
Much appreciate your oooparatlon. If thtra 1s anything alsa you would Ilka to shaN with 
us about cost studios, evaluation budgeting, or ways wo could further assist In the 
improvement of evaluation practice,, please use this page to tell us. 



Plaasa use the enclosed envelope to return this completed questionnaire to Or. Nick L. 
Smith, Research on Evaluation Program, Northwest Regional Educational Laboratory, 300' 
S. U. Sixth Avenue, Portland, OR 97204. For Inquiries, call 1-800-547-6339. 
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Northwest Regional E Jucational Laboratory 
300 S.W. Sixth Avenue 
Portland. Oregon 97204 
Telephone (503) 248-6800 
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